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[57] 



ABSTRACT 



Apparatus and a method of operation manipulates the dis- 
play of an object on the display surface of a display device 
in accordance with sensing touching contact on a touch 
panel, superimposed on the display surface and simulating 
movement of the displayed object in accordance with the 
characteristics of the touching contact on the touch panel 
adjacent to the displayed object image. Further, object data 
defining the display of the object image for each of a 
plurality of different states of the object, display information 
specifying the shape and physical properties of the object 
and the current display position thereof and file information 
relating to the stored data are stored and accessed for 
interpreting the simulated manipulation of the object by the 
characteristics of the touching contact therewith, for corre- 
spondingly manipulating and displaying the thus manipu- 
lated object image. 

42 Claims, 11 Drawing Sheets 
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APPARATUS FOR MANIPULATING AN To achieve the above and other objects, the present 

OBJECT DISPLAYED ON A DISPLAY invention provides a touch screen, a plurality of data files, 

DEVICE BY USING A TOUCH SCREEN display information storage means and display control 

means. 

This application is a continuation, of application number 5 i n the apparatus for manipulating an object displayed on 

07/953,101, filed Sep. 29,1992, now abandoned. a display device, the touch screen, which is a transparent 

BACKGROUND OF THE INVENTION panel and is mounted on the display surface, or screen, of a 

display device and is sensitive to the touch, e.g., of a finger 

1. Field of the Invention 0 f a human operator, outputs touch screen information 
The present invention relates to an apparatus for mampu- 10 representing the motion of the body. The plurality of data 

lating an object displayed on a display device by using a files store object data for displaying the object in different 

touch screen. states. The display information storage means stores object 

As use of computer systems for data processing has information including at least an object type which specifies 

become widespread in recent years, more and more users are the shape and physical properties of the object, display 

being required to handle same, i.e., to input data and 15 position information which specifies a position where the 

converse with data processors such as work stations and object is displayed on the display device, file information 

personal computers. A vast variety of application programs which specifies the size and location of a part of the object 

are available for recent data processors and even a compli- data stored in one of said plurality of data files, and a file 

cated application can be processed by using such application name which specifies one of said plurality of data files. The 

programs in combination. However, it is a great problem that 20 display control means recognizes a manipulation to be 

such data processors are very difficult to handle, especially conducted on the object, based on the touch screen infor- 

to manipulate an object displayed on a display device, for mation from the touch screen and on the object information 

those who have less than a substantial knowledge of com- included in the display information storage means, and 

puters. displays the object on the display device in accordance with 

Therefore, an apparatus for manipulating an object dis- 25 me aforesaid recognition, 

played on a display device, which is easily handled even for no tot nccrDiimnM nc toc rm aiutktpc 

a person who has no special knowledge of computers, is in BRIEF DESCRIPTION OF THE DRAWINGS 

great demand. FIG. 1 illustrates a computer system with a conventional 

2. Description of the Related Art 3Q user interface; 

FIG. 1 illustrates a computer system with a conventional FIG. 2 is a block diagram of the configuration of a touch 

user interface. screen-equipped workstation, in which the present invention 

A computer system with a conventional user interface is implemented; 

consists mainly of a central processing unit (CPU) 4, a main FIG. 3 is a schematic diagram illustrating the principle of 

memory 5, a keyboard/mouse 2, a frame memory 6 and a 35 the present invention; 

hard disk interface 71, which are interconnected via a system fig. 4(a) is a display information table; 

bus, and also a hard disk 7 and a display unit 3, which are nr /i/txn *.* .u c * r * u - r 

, . . • *u u j j- i ■ * _r j *l FIG. 4(b) illustrates the format of touch-screen mfonna- 

connected to system bus via the hard disk interface and the ^ on> 

frame memory 6, respectively. The main memory 5 stores a ' 

system control program and application programs which 40 FIG 5 15 a flowchart illustrating a pick manipulation; 

handle graphics processing, and provides a work area for use FIGS. 6(a) and 6(b) are related and interconnected dia- 

by the programs. The CPU 4 performs display operations grams illustrating and FIG. 6(c) is a table explaining a pick 

under control of the programs. The hard disk 7 stores a data manipulation; 

file for graphics to be displayed on the display unit 3. The FIG. 7(a) is a diagram illustrating and FIG. 7(b) is a table 

frame memory 6 stores a frame of picture (or object) data to 45 explaining a scroll manipulation; 

be displayed on the display unit 3. F1GSi S ( a ) and 8(b) are related diagrams illustrating and 

To manipulate an object displayed on a display unit 3 in FIG. 8(c) is a table explaining a push manipulation; 

the above system, an operator is required to input a FIGS. 9(a) and 9(b) are related diagrams illustrating and 

command, for manipulating the object by using a keyboard/ FIG. 9(c) is a table explaining a flip manipulation; 

mouse 2 or to select an icon (a symbolic representation of 50 pjQ 10(a) % a & illustrating and FIG. 10(6) is a 

a computer function), displayed on a display unit 3 by using ^ ^ a ^ manipulation; 

the keyboard/mouse 2, in order to command a desired ^„ . . .„ . , n . 

function. However, it is troublesome and annoying to use a ™ G ' 15 a ^! a S ram bating and FIG. 11(6) is a 

keyboard/mouse and icons and a person with less than a ex P lainin g a distort-restore manipulation, 

substantial knowledge of computers tends to be reluctant 55 Throughout the above-mentioned drawings, identical ref- 

even to touch a keyboard/mouse. ere nee numerals are used to designate the same or similar 

Therefore, it is a great problem that such data processors component parts, 

are very difficult to handle for those who have less than a DESCRIPTION OF THE PREFERRED 

substantial knowledge of computers. EMBODIMENTS 

SUMMARY OF THE INVENTION 60 nG . 2 is a block diagram of the configuration of a touch 

It is an object of the present invention to provide an screen-equipped workstation, in which the present invention 

apparatus which can easily manipulate an object displayed is implemented. 

on a display unit. In addition to the conventional system shown in FIG. 1, 

It is another object of the present invention to provide a 65 the system of FIG. 2, for implementing the present 

user interface with which a user can easily manipulate an invention, comprises an input-output (abbreviated to I/O) 

object displayed on a display unit. port 8, a touch screen controller 15 and a touch screen unit 
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1 with a touch screen 11. The touch screen controller 15, 
connected to the input-output port 8 through an RS-232C 
interface, controls the touch screen unit 1. The touch screen 
unit 1, which is sensitive, at a position (defined by X-Y 
coordinates) at which the touch screen 11 is touched and 
particularly to the pressure applied thereon when so touched, 
acts as a user interface that allows a user to send signals to 
a computer by touching that area thereon with an element, 
such as the finger, ... etc. 

FIG. 3 is a schematic diagram illustrating the principle of 
the present invention. 

For easy understanding of the principle, the input-output 
port 8, touch screen controller 15 and touch screen unit 1 
shown in FIG. 2 are represented in FIG. 3, in the composite, 
by the touch screen unit 1 and the frame memory 6 and 
display unit 3 and, as before-noted, the touch screen 11 
actually is a transparent panel which is mounted on the 
display surface, or screen, of the display unit 3 but is shown 
separately therefrom in FIG. 3 of FIG. 2 are represented in 
FIG. 3, in the composite, by the display unit 3. The hard disk 
interface 71 and hard disk 7 of FIG. 2 are represented in FIG. 
3, in the composite, by the hard disk 7. A system controller 
50, a touch discriminator 51, a display controller 52, which 
are programs stored in the memory 5 and executed by the 
CPU 4 (or may be constructed by hardware), and a display 
information table IT, stored in the main memory 5, control 
the display operations performed by the present invention. 

FIG. 4(a) shows a display information table. FIG. 4(6) 
shows touch screen information. 

The display information table IT, which is provided in the 
main memory 5 and corresponds to objects, includes object 
type information, display position information, file 
information, normal-display file name and special-state file 
name. The object type defines the type, including the shape, 
properties, circumstances, etc., of the object. The display 
position information defines the size of the object (width 
U W" and height "H"), and the position (top-left coordinates 
X,Y) and the angle at which the object is displayed on the 
display unit 1. The file information, which is used for an 
object which is so large in size that it requires scrolling to 
view the whole object, defines the size (width W, height H) 
of the whole object relative to the display screen size, and 
also the position (top-left coordinates X,Y) of the object 
portion being displayed on the display device, relative to the 
whole object whose data is stored in the normal file. The 
normal-display file name specifies a display data file where 
object data for displaying a normal state of the object is 
stored. The special-state file name specifies a display data 
file where object data for displaying a special state (e.g., 
turn-over indication of display color, used for selectively 
displaying an intermediate process step, or stage, in manipu- 
lating the object) of the object is stored. The special state can 
be displayed selectively for each manipulation. 

Touch-screen information 21, which is sent from the touch 
screen unit 1, includes a touch position (X-Y coordinates) 
where the touch screen 11 is touched and the pressure 
applied thereon. The touch -screen information 21 may 
include two sets of X-Y coordinates shown in FIG. 4(6) 
depending on the type of touch, one set for a thumb and 
another for an index finger when the two fingers pick the 
object on its opposite sides, for example. 
(1) The touch discriminator 51, based on the touch screen 
information 21 from the touch screen unit 1, discriminates 
the type of touch which the operator's finger has on the 
touch screen 11, that is, a touch type including, e.g., a 
"continuous touch start" and "continuous touch end" as 
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explained later. The touch discriminator 51 sends, to the 
system controller 50, the result of the touch discrimination 
performed thereby as a touch report 3R, which includes 
touch type and, according to the touch-screen information 

5 21, one or two sets of touch coordinates. 

Based on the touch report 3R from the touch discriminator 
51 and the display information table IT, the system control- 
ler 50 determines the type of manipulation which was 
conducted by the operator and, according to the 

10 determination, updates the display information table IT. 
Then, the system controller 50 sends, to the display con- 
troller 52, a display update request 4Q including "display 
update data" which reflects the updated contents of the 
display information table IT including display position 

15 information, filed information and normal display file name 
and special state file name. 

On receipt of the display update request 4Q from the 
system controller 50, the display controller 52 reads a 
display data file (including object data), specified by the file 

20 name, from the hard disk 7 and stores the data into the main 
memory 5. The display controller 52 then updates the object 
data in accordance with the display update data from the 
system controller 50 and loads the thus-updated object data 
into the frame memory 6 (FIG. 2) thereby to display the 

25 object on the display unit 3 (FIG. 2), as manipulated by the 
operator on the touch screen unit 1. 

Thus, the system of the present invention determines a 
manipulation to be conducted on the displayed object, based 
on the touch screen information 21 which results from an 

30 operator's touching the touch screen 11 and the display 
information table IT (see, FIG. 7(b)) which defines the 
object's shape, physical properties, display position, etc. The 
system then displays the object, according to the manipula- 
tion as thus determined and as intended by the operator, on 

35 the display unit 3. 

(2) Pick manipulation (see FIGS. 5 and 6(a) to 6(c)). 

A pick manipulation is conducted in such a way that an 
object is picked up at a position on the display surface of the 
display unit 3 and placed at another position thereon. 

40 FIG. 5 is a flowchart illustrating a pick manipulation. 
FIGS. 6(a) and 6(b) are related and interconnected dia- 
grams illustrating, and FIG. 6(c) is a table explaining, a pick 
manipulation. 

A pick manipulation is carried out according to the 
45 following steps (Sl-nS8) in FIG. 5: 

(51) The system controller 50 receives a touch report 3R 
including two sets of touch coordinates, from the touch 
discriminator 51. 

(52) The system controller 50 (FIG. 3) checks whether the 
50 object-finger relation is a pick manipulation, as shown in 

FIG. 6(a), based on the touch report 3R and contents of the 
display information table IT shown in FIG. 6(c). When the 
relation is not a pick manipulation, the system controller 50 
checks the touch report 3R for other possible manipulations. 

55 (S3) When the relation is a pick manipulation, the system 
controller 50 sends a display update request 4Q including 
"display update data", commanding that the special-state file 
(turn-over indication) be displayed at the position specified 
by the display information table IT. 

60 (S4) The system controller 50 receives a touch report 3R. 

(55) The system controller 50 determines whether the 
touch report 3R includes a "continuous touch end", which 
occurs when the finger-object relation is as shown in FIG. 
6(b). When a "continuous touch end" is reported, the opera- 

65 tion goes to step (S8). 

(56) Otherwise, the system controller 50 updates the 
display position information "coordinates (X, Y)" of the 
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display information table IT so that the object is positioned loads it in the main memory 5. The display controller 52 then 

between the two fingers of the operator. transfers only the part of the display file, as specified by the 

(57) The system controller 50 sends display update file information "display position X" of the display infor- 
request 4Q to the display controller 52, commanding that the mation table IT, from the main memory 5 to the appropriate 
special-state file be displayed according to the display 5 location of the frame memory 6. 

information table IT, as updated, and returns to step (S4). In the same way, the system controller 50 sends a display 

(58) When "continuous touch end" is reported by a touch update request 4Q to the display controller 52 every time it 
report 3R, the system controller 50 sends a display update receives a touch report 3R. 

request 4Q to the display controller 52, commanding that the When another "continuous touch" is reported before the 

normal-display file be displayed at the position specified in scroll currently in progress comes to a stop, a new scroll can 

the display information table IT. start from this point and at the first speed described above. 

The following manipulations are carried out in the same (4) Scroll-stop manipulation (see FIGS. 7(a) and 7(b)). 

way as described in the above flowchart of the pick manipu- FIG. 7(a) is a diagram illustrating, and FIG. 1(b) is a table 

lation. explaining, a scroll manipulation. 

(3) Scroll manipulation (see FIGS. 7(a) and 1(b)). When a touch position given by a touch report 3R is the 

A scroll manipulation is conducted in such a way that an 15 same as or up to approximately 5 dots apart from the 

object, extending outside of the display surface of the position of the scrolling currently in progress, the system 

display unit 3, is moved into and out of the display surface. controller 50 doubles the frequency with which display 

FIG. 7(a) is a diagram illustrating, and FIG. 1(b) is a table update request 4Q are sent to the display controller 52, in 

explaining, a scroll manipulation. order to put an end to the scrolling. 

On determining that the operator's finger moves while 20 (5) Push manipulation (see FIGS. 8(a) to 8(c)). 

touching the touch screen 11, based on the touch screen A push manipulation is conducted in such a way that an 

information 21 from the touch screen unit 1, the discrimi- object is pushed on the display surface of the display unit 3. 

nator 51 sends, to the system controller 50, a touch report 3R FIG. 8(a) is a diagram illustrating, and FIG. 8(c) is a table 

as previously discussed including, e.g., "continuous touch explaining, a push manipulation. 

start" for the "touch type" and also, e.g., "coordinates (800, 25 The system controller 50 determines the type of a 

800)" for the touch position. As another touch screen infor- manipulation, based on the touch report 3R and contents of 

mation 21 comes in, the discriminator 51 sends a touch the display information table IT shown in FIG. 8(c). When 

report 3R including, e.g., "continuous touch in progress" and the manipulation is a push manipulation as shown in FIG. 

"coordinates (780, 800)" (i.e., the movement to the left by 8(a), the system controller 50 sends, to the display controller 

the finger as shown in FIG. 7(a), and thus from X-800 to 30 52, a display update request 4Q including display position 

X=780 while at a fixed Y=800). When the touch screen information, file information and normal display file name 

information 21 is not sent for more than 100 milliseconds, so that the object is displayed close to the finger position 

for example, the discriminator 51 sends a touch report 3R reported by the touch report 3R. The above display operation 

including "continuous touch end" and, e.g., "coordinates is repeated until a "continuous touch end" is reported by a 

(700, 800)" (i.e., the final X,Y coordinate as of the "touch 35 touch report 3R. 

end") to the system controller 50. (6) Push-while-rotate manipulation (see FIGS. 8(b) and 

When a "continuous touch start" is reported and the 8(c)). 

"object type" is defined as "out-screen" in the display A push-while-rotate manipulation is conducted in such a 

information table IT as shown in FIG. 1(b), the system way that an object is pushed at a position off its center (or 

controller 50 recognizes the manipulation as a scroll and the 40 the center of gravity shown as X in FIG. 8(b)) and it moves 

object as a large one extending beyond the display screen. rotating on the display surface of the display unit 3. 

Then, the system controller 50 determines the speed at The system controller 50 determines the type of a 

which the finger has moved from right to left, for example, manipulation, based on the touch report 3R and contents of 

based on a change in the X-coordinate in terms of data, the display information table IT shown in FIG. 8(c). When 

between a touch report 3R and the following one. 45 the manipulation is a push -while-rotate manipulation as 

Depending on whether the finger has moved at a speed of shown in FIG. 8(6), the system controller 50 sends, to the 

more (high-speed) or less (normal-speed) than 20 dots, for display controller 52, display update requests 4Q with the 

example, the object display position on the display screen is angle of rotation increasing by 2 degrees, i.e., while increas- 

scrolled initially at corresponding intervals of 100 or 500 ing the angle in the display information table IT shown in 

milliseconds, respectively. Then, the interval, at which the 50 FIG. 8(c). 

display update request 4Q is sent to the display controller 52, The display controller 52 reads the display file from the 

is increased by a factor of 1.5 at each touch report 3R and, hard disk and loads the data in the main memory 5, rotates 

when the interval reaches 2 seconds, the scrolling is stopped. the object by the specified angle and with the left-top 

Practically, the display position area of the screen is so coordinates (X, Y) as a rotational center, as specified by the 

controlled that it starts scrolling at the appropriate speed, as 55 display update request 4Q, and transfers the data, with the 

above-mentioned, after the operator's finger has moved a object rotated, from the main memory 5 to the frame 

distance of 4 dots or more. That is, on recognizing that the memory 6. 

finger has moved by that distance, the system controller 50 (7) Flip manipulation (see FIGS. 9(a) to 9(c)). 

updates the file information "display position X" of the A flip manipulation is conducted in such a way that an 

display information table IT so that the object is displayed, 60 operator's finger flips an object or touches (i.e., impacts) the 

shifted to the left by 10 dots, for example. The, it sends, to object from a remote position at a high speed on the touch 

the display controller 52, a display update request including screen 11 with a corresponding result on the display surface 

display position information, file information and normal of the display unit 3. 

display file name from the display information table IT, as FIG. 9(a) is a diagram illustrating, and FIG. 9(c) is a table 

updated. 65 explaining, a flip manipulation. 

The display controller 52 reads from the hard disk 7 a When a touch report 3R is input from the touch discrimi- 

display file specified by the normal display file name and nator 51, the system controller 50 discriminates the type of 
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the manipulation, based on the touch report 3R and the 
contents of the display information table IT shown in FIG. 
9(c). When the manipulation is a flip manipulation as shown 
in FIG. 9(a), the system controller 50 obtains the finger 
speed based on the touch report 3R and also obtains the 5 
object speed (i.e., the respective intervals at which display 
update requests 4Q are sent to the display controller 52), in 
the same way as described in item (3). The system controller 
50 sends display update requests 4Q to the display controller 
52, while updating the display position information left-top 10 
coordinates (X, Y) of the display information table IT so 
that the object moves in the direction the finger moves. The 
system controller 50 stops moving the object when the 
above-mentioned interval reaches 2 seconds. 
(8) Flip-under-gravity manipulation (see FIGS. 9(b) and 15 
9(c)). 

A flip-under-gravity manipulation is conducted in such a 
way that an object, which is subjected to gravity, is flipped 
by a finger on the touch screen 11 and with a corresponding 
result on the display surface of the display unit 3. 20 

FIG. 9(a) is a diagram illustrating, and FIG. 9(c) is a table 
explaining, a flip manipulation. 

When the finger manipulation is a flip as in the above item 

(8) and the display information table IT defines the object 
type as "gravity" meaning that the object is subjected to 25 
gravity, for example, the object moves while receiving a 
combination of the forces of inertia and gravity, as shown in 
FIG. 9(b). Therefore, the system controller 50 sends display 
update requests 4Q to the display controller 52, while 
updating the display position information left-top coordi- 30 
nates (X, Y) by adding a value to the Y-coordinate of the 
display information table IT. The value is represented by 2 

to the Nth power (N: the number of display update requests 
4Q which are sent). In this case, too, the system controller 
50 stops moving the object when the above-mentioned 35 
interval reaches 2 seconds. 

(9) Roll manipulation (see FIGS. 10(a) and 10(b)). 

A roll manipulation is conducted in such a way that a 
Tollable object is rolled by the operator's finger on the touch 
screen 11 and with a corresponding result on the display 40 
surface of the display unit 3. 

FIG. 10(a) is a diagram illustrating, and FIG. 10(b) is a 
table explaining, a roll manipulation. 

When a touch report 3R is input from the touch discrimi- 
nator 51 and the display information table IT defines the 45 
object type as "tollable", meaning that the object is con- 
structed such that it rolls when pushed by a finger, like a 
globe or a cylinder and as shown in FIG. 10(a), the system 
controller 50 sends display update requests 4Q to the display 
controller 52, while updating the display position infonna- 50 
tion left-top coordinates (X, Y) of the display information 
table IT so that the object moves a distance 10 percent 
behind the distance moves on the display screen and in the 
direction the finger moves. The system controller 50 sends 
display update request 4Q to the display controller 52 until 55 
"continuous touch end" is reported from the touch discrimi- 
nator 51. 

(10) Distort-restore manipulation (see FIGS. 11(a) and 11(6) 
)• 

A distort-restore manipulation is conducted in such a way 60 
that an elastic object is pressed by a finger on the display 
surface of the display unit 3. 

FIG. U(a) is a diagram illustrating, and FIG. 11(6) is a 
table explaining, a distort-restore manipulation. 

When a touch report 3R is input from the touch discrimi- 65 
nator 51 and the display information table IT defines the 
object type as "elastic", meaning that the object can be 
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distorted and restored according to a pressure applied 
thereon by a finger and as shown in FIG. 11(a), the system 
controller 50 calculates an amount of disto rtipnaof the object 
based on the pressurejapofterh^v-tbe ^Such ^report 3R. It 
stores in the display information table IT, a special-stale file 
name specifying a corresponding one of the special-state 
files (for displaying a distorted state of the object in a 
turn-over indication) corresponding to the amount of distor- 
tion calculated. Then, the system controller 50 sends a 
display update request 4Q to the display controller 52, 
commanding that the special-state file be displayed at the 
current display position. The above operation is repeated as 
necessary and when a "continuous touch end" is reported by 
a touch report 3R, the system controller 50 sends a display 
update request 4Q (with a normal display file name 
specified) to the display controller 52, commanding that a 
normal display file (normal indication) be displayed at the 
current display position. A plurality of special-state files are 
provided in the hard disk 7, corresponding to the amount of 
distortion of the object which results from the pressure 
applied on the touch screen 11. 

As is apparent from the above description, the present 
invention regards a display screen as a visual space. It 
defines conditions and physical properties of an object (e.g., 
weight, hardness, frictional resistance, center of gravity) in 
the display information table IT. It also receives touch 
screen information 21, indicating a finger-touched position 
and pressure, as input from a touch screen unit 1. Based on 
the touch screen information 21 and the display information 
table IT, the present invention determines a manipulation to 
be conducted on the displayed object, e.g, scrolling, picking, 
pushing, rolling, distorting the object on the display surface 
of the display unit 3. Thus, the present invention allows a 
user to manipulate an object displayed on a display device 
quite easily, even when the user has little knowledge of 
computers. 

What is claimed is: 

1. An apparatus affording simulated manipulation of an 
object in accordance with an image of the object displayed 
on a display surface of a display device, comprising: 
a touch panel, mounted in superimposed relationship on 
the display surface of the display device and made of a 
transparent material through which the object image, 
displayed on the display surface, is transmitted, the 
touch panel being sensitive to characteristics of a 
touching contact on the touch panel adjacent the dis- 
played object image, the characteristics including at 
least coordinate positions of, and time-based changes 
in, the adjacent touching contact, and outputting cor- 
responding touch panel information representing the 
characteristics of the adjacent touching contact on the 
touch panel; 

a plurality of data files which respectively store object 
data defining the display of the object image on the 
display surface of the display device in each of a 
plurality of different states of the object; 

display information storage means for storing a plurality 
of different types of object information, including at 
least: 

object information specifying the shape and physical 
properties of the object, and 

display position information specifying the position on 
the display surface of the display device at which the 
object image currently is displayed; and 
display control means, responsive to the touch panel 

information output by the touch panel, for recognizing 

the simulated manipulation of the object image and, 
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further, responsive to the object data and the object 
information for the displayed object image and the 
recognized, simulated manipulation of the object, for 
correspondingly manipulating and displaying the 
manipulated object image on the display surface of the 5 
display device. 

2. An apparatus as recited in claim 1, wherein the display 
control means is responsive to, and recognizes as a pick 
manipulation, touch panel information representing touch- 
ing contact on the touch panel at first and second positions 10 
respectively adjacent first and second, opposite sides of the 
object at an initial display position and further to touch panel 
information representing continuing, coordinated movement 

of the first and second touching contact positions from the 
initial and to a further position, displaced by a distance from 15 
the initial position, for correspondingly moving the object 
image display by a corresponding said distance, from the 
initial position and to a displaced display positions, on the 
display surface. 

3. An apparatus as recited in claim 1, wherein; 20 
the display control means is responsive to, and recognizes 

as a push manipulation, touch panel information indi- 
cating both touching contact on the touch panel at a 
central position of the displayed object and also con- 
tinuing movement of the touching contact by a distance 25 
from an initial position to a displaced position, simu- 
lating movement of the displayed object and the stop- 
ping of such movement while maintaining touching 
contact with the displayed object and, in response to the 
recognized push manipulation, correspondingly moves 30 
the displayed object on the display surface by a corre- 
sponding distance to a displaced display position and 
then fixes the display of the displayed object at the 
displaced display position. 

4. An apparatus as recited in claim 1, wherein the display 35 
control means is responsive to, and recognizes as a push- 
while-rotate manipulation, touch panel information repre- 
senting both touching contact on the touch panel at a 
position displaced from the center of the displayed object 
and also movement of the touching contact by a distance 40 
from an initial position to a displaced position and the 
ending of the movement while maintaining touching contact 
with the object and, in response to the recognition of the 
push-while-rotate manipulation, correspondingly both 
moves and rotates the object image display from the initial 45 
display position and through a corresponding distance to a 
displaced display position and, then, fixes the display of the 
object image at the displaced display position. 

5. An apparatus as recited in claim 1, wherein the display 
control means is responsive to, and recognizes as a flip 50 
manipulation, touch panel information representing move- 
ment of the touching contact on the touch panel from an 
initial position displaced from the object image display at a 
speed higher than a predetermined speed and into simulated 
impacting contact with the displayed object image and, in 55 
response to the recognized flip manipulation, correspond- 
ingly both moves the object image display by a distance, 
which is proportional to the speed of movement of the 
touching contact on the touch panel when simulating contact 
with the object display image at the initial display position, 60 
and in a direction, which corresponds to the direction of the 
touching contact movement at the simulated contact with the 
object image at the initial display position. 

6. An apparatus as recited in claim 1, wherein the display 
control means is responsive to, and recognizes as a flip- 65 
under-gravity manipulation, touch panel information repre- 
senting movement of the touching contact on the touch panel 
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at a speed greater than a predetermined speed, from an initial 
position displaced from the initial display position of the 
object image and into simulated impacting contact with the 
displayed object image at an initial object image display 
position and, further, is responsive to object information of 
the displayed object image specifying the object type as 
being subject to the effects of gravity, for correspondingly 
moving the object image display from the initial object 
image display position by a distance proportional to the 
speed with which the touching contact simulates contact 
with the object image at the initial object image display 
position and along a parabolic path of movement corre- 
sponding to the travel of the object when impacted into 
movement and subjected to the effects of gravity. 

7. An apparatus as recited in claim 1, wherein: 

the touch panel is sensitive to the level of pressure of the 
touching contact thereon; and 

the display control means is responsive to, and 
recognizes, as a distort-restore manipulation, touch 
panel information representing touching contact on the 
touch panel simulating a pressure-applying contact on 
the object and the pressure level of the pressure- 
applying touching contact and the release of the 
pressure-applying contact and, based on object infor- 
mation specifying the object type as being elastic, for 
correspondingly altering the configuration of the object 
image from that as initially displayed, in accordance 
with, respectively, distorting the displayed object 
image by an amount proportional to the pressure level 
of the simulated pressure-applying contact and the 
elasticity of the object and subsequently restoring the 
initial configuration of the object image released by the 
pressure of the touching contact. 

8. An apparatus as recited in claim 1, wherein the display 
information storage means further comprises means for 
storing file information which specifies a position of the 
object image portion being displayed on the display device, 
relative to the whole object image whose data is stored in 
one of said plurality of data files. 

9. An apparatus as recited in claim 8, the display surface 
having a fixed perimeter and corresponding display 
dimensions, wherein: 

the display control means is responsive to, and recognizes 
as a scroll manipulation, touch panel information indi- 
cating touching contact on the touch panel and 
associated, simulated movement of the displayed object 
by the movement of the touching contact while main- 
taining the adjacent touching contact relationship with 
the displayed object image, and further is responsive to 
object information, specifying that the displayed object 
is an "out-screen" object type and therefore has dimen- 
sions exceeding the dimensions of the display surface 
of the display device, for controlling, and thereby 
producing, a scrolling display of the object image on 
the display surface of the display device. 

10. An apparatus as recited in claim 9, wherein the display 
control means is responsive to, and recognizes as the end of 
the scroll manipulation, touch panel information from the 
touch panel indicating the discontinuation of the continuing, 
coordinated movement of the touching contact and corre- 
spondingly the discontinuation of the simulated, continuing 
movement of the displayed object image and, in response 
thereto fixes the object image display at the current display 
position thereof on the display surface of the display device. 

11. A method for simulating manipulation of an object in 
accordance with a touching contact adjacent an image of the 
object displayed on a display surface of a display device 
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employing a touch panel, mounted in superimposed rela- 
tionship on the display surface of the display device and 
made of a transparent material through which the object 
image, displayed on the display surface, is transmitted, the 
method comprising: 

sensing characteristics of a touching contact on the touch 
panel adjacent the displayed object image, the charac- 
teristics including at least coordinate positions of, and 
time-based changes in, the adjacent touching contact, 
and producing corresponding touch panel output infor- 
mation representing the characteristics of the adjacent 
touching contact on the touch panel; 
storing object data defining the display of the object 
image on the display surface of the display device in 
each of a plurality of different states of the object in a 
plurality of data files; 
storing display information comprising a plurality of 
different types of object information, including at least: 
object information specifying the shape and physical 

properties of the object, and 
display position information specifying the position on 
the display surface of the display device at which the 
object image currently is displayed; and 
responding to the touch panel information output by the 
touch panel for recognizing the simulated manipulation 
of the object and, further, to the object data and the 
object information for the displayed object image for 
correspondingly manipulating and displaying the 
manipulated object image on the display surface of the 
display device. 

12. A method as recited in claim U, further comprising: 
responding to, and recognizing as a pick manipulation, 

touch panel information representing touching contact 
on the touch panel at first and second positions respec- 
tively adjacent first and second, opposite sides of the 
object at an initial display position and further to touch 
panel information representing continuing, coordinated 
movement of the first and second touching contact 
positions from the initial and to a further position, 
displaced by a distance from the initial position, for 
correspondingly moving the object image display by a 
corresponding said distance, from the initial position 
and to a displaced display position, on the display 
surface. 

13. A method as recited in claim 11, further comprising: 
responding to, and recognizing as a push manipulation, 

touch panel information indicating both touching con- 
tact on the touch panel at a central position of the 
displayed object and also continuing movement of the 
touching contact by a distance from an initial position 
to a displaced position, simulating movement of the 
displayed object and the stopping of such movement 
while maintaining touching contact with the displayed 
object and, in response to the recognized push 
manipulation, correspondingly moving the displayed 
object on the display surface by a corresponding dis- 
tance to a displaced display position and then fixing the 
display of the displayed object at the displaced display 
position. 

14. A method as recited in claim U, further comprising: 
responding to, and recognizing as a push-while-rotate 

manipulation, touch panel information representing 
both touching contact on the touch panel at a position 
displaced from the center of the displayed object and 
also movement of the touching contact by a distance 
from an initial position to a displaced position and the 
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ending of the movement while maintaining touching 
contact with the object and, in response to the recog- 
nition of the push-while-rotate manipulation, corre- 
spondingly moving and rotating the object image dis- 
play from the initial display position and through a 
corresponding distance to a displaced display position 
and then fixing the display of the object image at the 
displaced display position. 

15. A method as recited in claim U, further comprising: 
responding to, and recognizing as a flip manipulation, 

touch panel information representing movement of the 
touching contact on the touch panel from an initial 
position displaced from the object image display at a 
speed higher than a predetermined speed and into 
simulated impacting contact with the displayed object 
image and, in response to the recognized flip 
manipulation, correspondingly both moving and rotat- 
ing the object image display by a distance, which is 
proportional to the speed of movement of the touching 
contact on the touch panel when simulating contact 
with the object display image at the initial display 
position, and in a direction, which corresponds to the 
direction of the touching contact movement at the 
simulated contact with the object image at the initial 
display position. 

16. A method as recited in claim U, further comprising: 
responding to, and recognizing as a flip-under-gravity 

manipulation, touch panel information representing 
movement of the touching contact on the touch panel at 
a speed greater than a predetermined speed, from an 
initial position displaced from the initial display posi- 
tion of the object image and into simulated impacting 
contact with the displayed object image at an initial 
object image display position and, further, responding 
to object information of the displayed object image 
specifying the object type as being subject to the effects 
of gravity, and correspondingly moving the object 
image display from the initial object image display 
position by a distance proportional to the speed with 
which the touching contact simulates contact with the 
object image at the initial object image display position 
and along a parabolic path of movement corresponding 
to the travel of the object when impacted into move- 
ment and subjected to the effects of gravity. 

17. A method as recited in claim U, further comprising: 
responding to, and recognizing, as a distort-restore 

manipulation, touch panel information representing 
touching contact on the touch panel simulating a 
pressure-applying contact on the object and the pres- 
sure level of the pressure-applying touching contact 
and the release of the pressure-applying contact and, 
based on object information specifying the object type 
as being elastic, correspondingly altering the configu- 
ration of the object image from that as initially 
displayed, in accordance with, respectively, distorting 
the displayed object image by an amount proportional 
to the pressure level of the simulated pressure-applying 
contact and the elasticity of the object and subsequently 
restoring the initial configuration of the object image 
released by the pressure of the touching contact. 

18. A method as recited in claim 11 wherein said respond- 
ing further comprises: 

providing, as said data files, a normal state display file 
which stores object data for displaying the normal state 
of the object, absent any simulated manipulation 
thereof, and plural sets of special state files respectively 
corresponding to the plural types of simulated manipu- 
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lations of the object, each set storing initial and suc- 
cessive object data for displaying initial and successive 
special states of the object corresponding to successive 
conditions of the object resulting from the respective, 
simulated manipulation; 5 

generating a touch report, in accordance with the touch 
panel information output by the touch panel in response 
to the touching contact on the touch panel adjacent a 
displayed object, the touch report identifying the object 
type of the displayed object and the recognized, simu- 1Q 
lated manipulation of the object; 

in response to the touch report and the recognized, simu- 
lated manipulation provided thereby, accessing the 
stored display information including the special state 
file corresponding to both the object type and also the 15 
recognized, simulated manipulation of the object, and 
updating the stored display information with the initial 
object data of the special state file, 

displaying the object in accordance with the updated 
display information at the display position defined by 20 
the stored display position information; 

generating a further touch report and determining whether 
the simulated manipulation has continued or has ended, 
and: 

(a) if the simulated manipulation has ended, accessing 2 5 
the normal display file and updating the stored 
display information in accordance with the object 
data of the normal display file and displaying the 
object at the display position specified by, and in 
accordance with the thus updated object information, 30 
and 

(b) if the simulated manipulation has continued, access- 
ing the special state file for obtaining therefrom the 
next successive object data corresponding to the next 
successive special state of the object, updating the 35 
stored display information with the thus obtained 
next successive object data, and displaying the object 

in accordance with the thus updated and stored 
display information and repeating the present step 
(b) for each successive, further touch report from 40 
which it is determined that the simulated manipula- 
tion has continued. 

19. A method as recited in claim 18 wherein the simulated 
manipulation of the object is a pick manipulation. 

20. A method as recited in claim 11, further comprising 45 
storing file information which specifies a position of the 
object image portion being displayed on the display device, 
relative to the whole object image of the stored object data. 

21. A method as recited in claim 20, the display surface 
having a fixed perimeter and corresponding display 50 
dimensions, further comprising: 

responding to, and recognizing as a scroll manipulation, 
touch panel information indicating touching contact on 
the touch panel and associated, simulated movement of 
the displayed object by the movement of the touching 55 
contact while maintaining the adjacent touching con- 
tact relationship with the displayed object image, and 
further is responsive to object information, specifying 
that the displayed object is an "out-screen" object type 
and therefore has dimensions exceeding the dimensions 60 
of the display surface of the display device, for 
controlling, and thereby producing, a scrolling display 
of the object image on the display surface of the display 
device. 

22. A method as recited in claim 21, further comprising: 65 
responding to, and recognizing as the end of the scroll 

manipulation, touch panel information from the touch 
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panel indicating the discontinuation of the continuing, 
coordinated movement of the touching contact and 
correspondingly the discontinuation of the simulated, 
continuing movement of the displayed object image 
and, in response thereto, fixing the object image display 
at the current display position thereof on the display 
surface of the display device. 

23. Apparatus for simulating manipulation of an object in 
accordance with touching contact adjacent an image of the 
object displayed on a display surface of a display devices 
comprising: 

a memory storing, for each object image selectable for 
display on the display surface of the display device, 
object information specifying a type of the object and 
the shape and physical properties of the object; 

a touch panel, mounted in superimposed relationship on 
the display surface of the display device and made of a 
transparent material through which an object image, 
selected and displayed on the display surface, is 
transmitted, coordinate positions being defined on the 
touch panel corresponding to coordinate positions of 
the display surface of the display device and the touch 
panel being sensitive and responsive to a touching 
contact thereon, adjacent a displayed object image, for 
outputting corresponding touch panel information, the 
touch panel output information comprising coordinate 
positions of, and time-based changes in, the adjacent 
touching contact; and 

display control means, responsive to the touch panel 
output information, for controlling the display of an 
object image, including changing the display of the 
object image in response to a change in the coordinate 
positions of the adjacent touching contact on the touch 
panel adjacent to the displayed object image and in 
accordance with the stored object information. 

24. An apparatus as recited in claims 23, wherein the 
display control means further is responsive to touch panel 
output information indicating time-based changes in the 
coordinate positions of the touching contact for controlling 
the displayed object image to simulate corresponding move- 
ment thereof. 

25. A method for manipulating an image of an object 
displayed on a display surface of a display device, compris- 
ing: 

storing, for each object image selectable for display on the 
display surface of the display device object information 
specifying a type of the object; 

defining a coordinate system on the display surface of the 
display device; 

displaying an image object image on the display surface 
of the display device, including controlling the coor- 
dinate position of the displayed object image; 

sensing touching contact in relation to the display surface 
of the display device adjacent to the displayed object 
image and outputting corresponding adjacent touching 
contact information including the effective coordinate 
positions of the adjacent touching contact and time- 
based changes therein; and 

controlling the display of the object image responsive to 
the touch panel output information, including changing 
the display of the object image, in response to a change 
in the coordinate positions of the touching contact on 
the touch panel adjacent to the displayed object image 
and in accordance with the stored object information. 

26. A method as recited in claim 25, further comprising 
responding to touch panel output information indicating 



06/17/2003, EAST Version: 1.03.0002 



5,8- 

15 

lime-based changes in the coordinate positions of the touch- 
ing contact for controlling the displayed object image to 
simulate corresponding movement thereof. 

27. An apparatus affording simulated manipulation of an 
object in accordance with an image of the object displayed 
on a display surface of a display device, comprising: 

a display device displaying an image of an object; 

an input device responsive to a touching contact adjacent 
a position of the displayed object image as displayed by 
the display device, the adjacent touching contact simu- 
lating a manipulation of the object, and outputting 
corresponding touch information; 

a characteristics detector receiving and detecting, from 
the received touch information, characteristics of the 
adjacent touching contact, the characteristics including 
the position of the adjacent touching contact and time- 
based changes of the position of the adjacent touching 
contact, and recognizing therefrom the corresponding 
object manipulation simulated thereby; 

a storage unit storing object information specifying a type 
of the displayed object image; and 

a display controller changing the display of the object 
image in accordance with the stored object information 
for the displayed object image and the recognized, 
simulated object manipulation. 

28. An apparatus according to claim 27, wherein: 

in response to detected characteristics indicating that two 
touching contacts are located at opposite sides of the 
object image and move and stop with a distance 
therebetween, said display controller controls the dis- 
play of the object image on the display device so that 
the object moves on a display surface of the display 
device from where the two touching contacts are 
located at opposite sides of the object image to where 
the two touching contacts stop with the distance ther- 
ebetween. 

29. An apparatus according to claim 27, wherein: 

in response to the detected characteristics indicating that 
the touching contact position is located at the object 
and moves while keeping in contact with the object and 
the object is sufficiently large relative to, and extends 
beyond, the display or only a portion of the object 
image is displayed on the display device, the display 
controller displays the object by scrolling the object 
image on the display device. 

30. An apparatus according to claim 29, wherein: 

in response to the detected characteristics indicating that 
a moving, touching contact stops moving, the display 
controller displays the object image on the display 
device so that the scrolling of the object image stops on 
the display. 

31. An apparatus as recited in claim 27, further compris- 
ing an information storage unit storing information speci- 
fying a position of a portion of the object image being 
displayed on the display device, relative to the whole object 
image. 

32. An apparatus according to claim 27, wherein: 

in response to the detected characteristics indicating that 
the moving, touching contact is located at a geometric 
center or at a center of gravity of the object and moves 
and stops while maintaining contact with the object, 
said display controller displays the object so that the 
object moves on a display surface thereof from where 
the touching contact is located at the geometric center 
or the center of gravity to where the moving, touching 
contact stops moving. 
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33. An apparatus according to claim 27, wherein: 
in response to the detected characteristics indicating that 
touching contact is located at a position off the geo- 
metric center or off the center of gravity thereof and 
5 moves and stops while maintaining contact with the 
object, said display controller displays the object so 
that the object moves while rotating on the display, 
from where the touching contact is located at a position 
to where the touching contact stops. 
io 34. An apparatus according to claim 27, wherein: 

in response to the detected characteristics indicating that 
touching contact touches the object, coming in a direc- 
tion from a position apart therefrom and at a speed 
higher than a predetermined speed, said display con- 
15 troller displays the object so that the object moves a 
distance in the direction and at a speed proportional to 
the speed with which the touching contact touches the 
object. 

35. An apparatus according to claim 27, wherein: 

20 said apparatus further comprises a display information 

storage unit storing object information specifying a 

type of the object as "rollable w ; and 
in response to the detected characteristics indicating that 

the touching contact touches the object image, moving 
25 and stopping while maintaining contact with the object, 

the display controller displays the object image in a 

rolling condition. 

36. An apparatus according to claim 27, wherein: 

the input device is sensitive to an amount of pressure 
30 applied thereon and, when the detected characteristics 
indicate the touching contact touches the object with an 
amount of pressure, the display controller displays the 
object image so that the object image varies in a degree 
of distortion and/or restoration according to the amount 
35 of pressure of the touching contact. 

37. An apparatus according to claim 27, wherein the input 
device is a panel, mounted with a display device, which is 
sensitive to the characteristics of touching contact on the 
panel 

40 38. An image display and manipulating apparatus, com- 
prising: 

a display device displaying an image of the object; 
an input device inputting a touching contact adjacent a 
45 position of the object image as displayed by the display 
device; 

a characteristics detector receiving and detecting, from 
the received touch information, characteristics of the 
adjacent touching contact, the characteristics including 
50 the position on the object image of the adjacent touch- 
ing contact and time based changes of the position of 
the adjacent touching contact, and recognizing there- 
from the corresponding object manipulation simulated 
thereby; 

55 a display information storage unit storing object informa- 
tion including at least an object type, which specifies 
physical properties of the object; and 
a display controller changing the display of the object 
image in accordance with the stored object information 
60 specifying physical properties of the object and the 
recognized simulated object manipulation. 
39. An apparatus manipulating a displayed image of an 
object, comprising: 

a display device displaying an image of the object; 
65 an input device inputting a touching contact adjacent a 
position of the object image displayed by the display 
device; 
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a characteristics detector receiving and detecting, from 
the received touch information, characteristics of the 
adjacent touching contact, the characteristics including 
a position of the adjacent touching contact on the object 
image and time based changes of the adjacent touching 5 
contact input by said input device; 

a display information storage unit storing object informa- 
tion including at least an object type, which specifies 
physical properties of the object, and position 
information, which specifies a position at which the 10 
object image is displayed on the display device; and 

a display controller changing the display of the object 
image in accordance with the stored object information 
and the detected characteristics. 

40. An apparatus simulating manipulation of an object in 15 
accordance with touching contact adjacent art image of the 
object displayed on a display surface of a display device, the 
apparatus comprising: 

a touch panel mounted in relation to a display surface of 2Q 
a display device and sensitive to characteristics of a 
touching contact adjacent the object image, the char- 
acteristics including at least a position of, and time- 
based changes in, the adjacent touching contact, for 
outputting corresponding touch panel information rep- 25 
resenting a simulated movement of the object respon- 
sive to the adjacent touching contact; 

a first display information storage means for storing 
object information including at least: 
an object type which specifies physical properties of the 30 
object, and 

position information which specifies the position where 
the object image is displayed on the display device; 
and 

a display controller recognizing a type of manipulation of 35 
the object image in accordance with the information 
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representing a simulated movement of the object, out- 
put by the touch panel, and the object type stored in the 
display information storage unit and correspondingly 
manipulating and displaying the object image on the 
display device. 

41. An apparatus as recited in claim 40, further compris- 
ing a second display information storage unit storing infor- 
mation specifying a position of a portion of the object being 
displayed on the display device, relative to the whole object. 

42. A display apparatus simulating manipulation of an 
object, comprising: 

a display device producing a display of an image of an 
object; 

a display information storage unit storing object 
information, including at least an object type which 
specifies physical properties of the object; 

a device responsive to touching contact thereon, adjacent 
to the displayed object image, simulating an actual 
touching contact with an actual object corresponding to 
the displayed object image; 

a detecting unit detecting characteristics of the adjacent 
touching contact including a position of the adjacent 
touching contact relatively to the displayed object 
image and time-based changes of the position of the 
adjacent touching contact and recognizing therefrom 
the corresponding object manipulation simulated 
thereby; and 

a display controller changing the display of the image of 
the object in accordance with the stored object infor- 
mation specifying Physical properties of the object and 
the recognized simulated object manipulation. 
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